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Measuring Our Progress with Real Tools: Why does 
technology work in some schools and not in others? 

Presentation outline by Bemajean Porter porterbj@edtechplanners.com 
Technology as a System Lever 

>- Technology magnifies and makes visible all that works in a school and all that needs to work 
better. Porter, 1994 

Over the years ... 

>- Training and Conducting audits in over 2500 schools 
>- Clark County Schools - Las Vegas, NV 

► Denver Public Schools - Colorado 

► DoDDs - Hanau Model Schools - Germany 

► Pinella County Schools - Florida 

► San Luis Obispo County Schools 

Scalability: Illinois NextSteps: Organizing Technology for Student Results 

► Sponsored by NCRTEC/NCREL with ISBE 

► State- wide toolkit for assessing technology’s impact 

► Six-day Internship Training 

>- Training 150+ educators to conduct technology and learning audits 

► Seven regional states shadowing project 

Asking BIG Questions of Our Technology Investments 

► What’s happening for ALL students? 

► What is the “added-value?” 

► What’s worth the dollars, energy and effort? 

What learning that we value would be impaired or impossible without the technology? 

Making IT Happen for ALL Students 

►Schools need a force-field analysis of what’s working and what needs to work better in the 
system to address the technology issues of scalability, equity and accountability. 

►Cannot move past initial 20-30% of the system adopting the innovation without the system 
making a concerted effort to adapt its culture and norms to enable the benefits bringing added- 
value. 

Stages of Staff Development - Guskey, ’’The Age of Our Accountability” JSD, 
Fall 1998 

► Note: Unless the system makes a concerted effort to address the issues occurring at Stage III, 
the gains in Stage I and II will dissipate over time. 

► Stage I: Attitudes 

► Stage II: Knowledge, skills and new attitudes 

► Stage III: Organizational support and change 

► Stage IV: Use of new knowledge and skills 

► Stage V: Student learning outcomes 
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Diffusion of Innovations - Rogers 

>- 8% Trailblazers (innovators) 

»- 17% Pioneers (early adopters) 

► 29% Settlers (early leaders) 

► 29% Stay-At-Homers (late leaders) 

► 17% Saboteurs (naysayers/resisters) 

Systems Alert! Assessing Pervasiveness! 

► Systems tolerate 25 -30% novelty - then the system tries to kill IT off! 

► When the percentages begin to reach at least 50% - You have early signs that the system is 
beginning to support, adapt and change itself in order to incorporate the benefits of the 
innovation. Lots of work ahead for collaborative leadership strategizing barriers! 

► When the percentages reach 80% or better - you have now established new norms in the 
organization 

First Order Change 

► Focused on technical tasks 

► Focused on getting people to do IT 

► Focused on efforts 

► Focused on broad, generalized learning goals 

Second Order of Change 

► Focused on re-culturing 

Focused on addressing equity / building critical mass 

► Focused on meaningful, “worth-the- money” student uses 

► Focused on added-value results 

► Focused on specific, measurable learning goals 

Organizing the System for Student Results 
Next Stop: Systems Thinking 

► Issues: Scalability, equity, and accountability 

► Implementation teams need to be “Cat Herders” 

► Organizing for Return On Investment (ROI) 

► Preparing for Second Order of Change 

Exp’s Four Cornerstones 

Each of the four questions have 6 Indicators. All four areas need equal attention to address 
technology’s issues of scalability, equity and accountability. There are over 75+ templates and 
tools organized to measure all indicators. Sample reports are organized into four chapters 
representing each of the four areas. See pdf file ETP’s Four Cornerstones. 

I. Readiness to Change 

II. Teaching and Learning 

III. System Capacity 

IV. Technology Deployment 
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Data Collecting is a BIG Job! 

► Identifying what you need to know 

► Ranking guide for each indicator 

► Qualities / Traits 

► Assessment tools for each indicator 

► Triangulation - three or more data sources to create a “thread of truth” that districts can 
trust enough to take action on! 

► Processes are needed to operationalize the frameworks into doable, usable tasks that provide 
practical information to school teams 

Example: Indicator 2: Leadership measures six traits in order to draw 
conclusions 

► Stewardship 

► Collegial strategizing 

► Change management 

► Support/expectations of staff 

► Professional skills/practices 

► Urgency 



Data in their Face - Going for “Catalytic Validity” 

► Educating and mobilizing the system 

► The degree to which the research (information) energizes participants toward knowing reality 
in order to transform it.” 

Studying Your Own School, 

1994 Gary Anderson et al. 

District Strategy Retreat - This process is critical to involve groups in owning 
the problems and the solutions 

► Member’s Check - provides a quality gate in reviewing the data 

► Creating Action Plans from the data reports 

Where’s the Flashlight? 

► Efforts vs Results - Michael Schmoker’s book, Results 

► Equipment efforts vs instructional results 

►Automating or re-culturing our schools as we know them today 

Tech Plan Analysis Tool - (see pdf file) 

► Section 1: Overall Technology Plan 

► Section 2: Analysis of Goals of Efforts or Results 

► Section 3: Analysis of Successfully Organizing Goals for Student Results 



Prepared by Bernajean Porter 9/11/00 



Page 3 



O 



5 



In the Beginning . . . 

>- Technology plans = purchase orders 

► 3-5 years ago “learning” became expected as part of federal / state funding 

»■ Problem — technology committees worked separately from curriculum groups 
>■ Many groups solved this dilemma with broad “fluff’ words to keep from invading teacher / 
curriculum domains 

Glorious Goals Found in State and District Plans - How do you measure 
these? 

► Enriching 

► Enhancing 

► Supporting standards 

► Creating life-long learners 

► Raising student achievement 

>• Integrating - how is it being defined? 



Integrating is . . . 

► Do something 

► Do anything ... 

► Just use it! 

Translated as . . . 

► Do what you can 

► Do what you want 

>• As long as you use it . . .for something! 

Sample Goals - try categorizing these for assessment instruments 

»- Students will be life-long learners in a technological society. 

>- Technology will be integrated throughout the curriculum. 

► Technology will enrich/enhance student achievement 

► Teacher, principals, and students will have opportunities to become comfortable with 
technology to improve all aspects of learning. 

ETP’s Spectrum of Technology and Learning Uses 

► Literacy Uses - telling technology stories about student learning 

► Adapting Uses -telling the same stories about student learning with new tools 

► Transforming Uses -telling new stories NOT possible or probable without the 

technology 

Mapping Learning Uses 

Item analysis of staff development workshop titles, teacher lesson plans, interviews and 
technology plans into the three categories of technology and learning uses. Generally what is 
being taught to the teachers is what is actually being implemented in the classrooms. Are you 
teaching what you want to happen in the classrooms for students? 
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The question is not what category you are supporting with technology uses - the real question is 
are you getting what you want? Is what you are getting worth the money and effort? 

What ETP is Learning. . . Schools are NOT Presently Organized for Results 

>■ Scoring “Accountability Readiness” - see www.edtechnlanners.com for on-line scoring of 
essential elements needed to organize for accountability. 

►What we have found in hundreds of schools conducting audits: 

► Schools are not presently organized for student results 

► Critical mass of systems are in first order of change 

► Technology use is optional NOT essential 

► Goals are focused on “efforts” rather than student RESULTS or benefits 

► Goals/indicators are NOT presently measurable 

► Baseline data for student results has NOT BEEN collected to chart progress 

► Data collection is mostly about technical availability and use 

► Learning evidence is presently more anecdotal than empirical 

► Funds ARE NOT allocated for data-based decisions / evaluation 

ETP has recently partnered with Phi Delta Kappa to train educators through 
six-day internships to acquire the skills and tools to assess and evaluation their 
technology’s impact. (See www.pdkintl.ore for details.) 

Skills do NOT Equal Practices 

Surveys alone cannot reveal what is really happening for students. Surveys are one-dimensional 
reportings. Sometimes they tend to represent “collective delusions” of what is happening. Need 
external evaluators to conduct observations, interviews, and focus groups in connection with 
surveys to determine what’s working and what needs to work better with actual practices. 

First/Second Order Evaluation 

► First Order — Efforts 

► Measuring the doing 

► What happened for adults and equipment? 

► What was completed/accomplished? 

► Second Order — Results 

*- Measuring the growth of benefits / changes 
>- What happened for students? 

*- What is the “added-value” NOT possible before the investment of 
dollars and the implementation work? 

How Does IT Really for Students? 

Classroom Observations: “Knowing the Culture Context” 

Walk-through Observations in Buildings - Help assess the following 

indicators: 

► Ubiquitous Access 

► Equitable Learning 

► Tool Capacity 

► Connectivity 
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Pedagogical Readiness 

► Instructional Practices 

► Student Work 

Interviews: The Real Story 

Leaders 

► Teachers 

► Parents 

► Community 

► Students 




Evaluating Student Products 



NCRTEC/N CREL in partnership w/ Education Tech Planners 

► How about using student products to deepen the evaluation of the impact of technology on 
student learning? 

► 70+ Comprehensive Assessment Tools 

► To determine the quality of content/craftsmanship of student work 

► To conduct a meta-analysis of the system use of technology 



Collaborative Development 

»- Sponsored a national think-tank team 



► Annette Lamb 
*- Lisa Holms 

*- Anita Dosaj 

► Bemajean Porter 
Critical friends 

*- Karen Bunting 
»- Lynell Burmark 
*- Jim Sydow 
►Tom Valentin 



Kristin Ciesemier 
Marilyn Jerde 
Bruce McKay 
Ian Jukes 

Carolyn McCullen 
Helen Hoffman 
Pete Denzin 



Student Products Help Assess the System’s Use of Technology (see pdf file 
SystemAssess/Student Products) 



► What are the expectations and results of student computer-based products? 

► Nine indicators score the system: 

► Standards-based Learning 

► Curriculum Linking 

► Cognitive Tasks 

>- Assessment Practices 
Content Achievement 

► Craftsmanship of Communication 
>- Authentic Tasks 

>- Instructional Uses / Practices 

► Added- value 
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Eight Trait Student Product Scoring Guides (see pdf file “Overview of Scoring 
Guides” and “Sample Scoring Guide”) 

► Part I: Content Communication 

► Preparation of Tasks 
»■ Content Knowledge 
>- Format/Organization 

Craftsmanship of Tool Use 

>■ Part II: Pathways of Communication 
>- Text Communication 
>- Image Communication 
. »• Voice / Sound Commimication 
>■ Design of Communication 

► Interactivity of Communication 

Five Uses of the Tools 

► 1. External evaluators * 

>- 2. Internal evaluators * 

► 3. Staff training * 

► 4. Peer collaborative scoring * 

► 5. Individual scoring 

► ^Collaborative assessment used in 1-4 Uses 

Classroom “Running Records” 

► Using Student Product Evaluation Tools for 

>■ Continuous assessment of student work 
*• Capacity to assess progress over time 

Grappling with Accountability: Resource Tools for Organizing and Assessing 
Technology for Student Results by Bemajean Porter 

► Field-tested 8 years 

► Do-it-yourself guide 

»- ETP’s Four Cornerstones Framework plus 

► 80+ digital tools / templates / sample reports 

>- Processes for data collection, project management, as well as for energizing and mobilizing 
the system to take action 



Ready or NOT! 

Herding those “CATS” 

>• It’s not going to be enough to know how to do IT . , . 

► we are going to have to be able to do IT even when we don’t know how! 



Prepared by Bernajean Porter 9/11/00 



Page 8 



O 



ERIC 



9 



For More Information: 



* 



www.edtechplanners.com — To download conference handouts, use all caps 
and the following key words - User ID = DOE Password = KIDS 

www.pdkintl.org - To request more information on workshops, on-line 
survey services, and comprehensive technology audits. 
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